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FOREWORD 


This  paper  was  prepared  as  a Technical  Note  to  support  the  "Future 
Planning"  session  of  the  Symposium  on  the  Future  of  Simulators  in  Skills 
Training.  The  symposium  was  held  as  a part  of  the  First  International 
Learning  Technology  Congress  and  Exposition,  sponsored  by  the  Society  for 


Applied  Learning  Technology,  Washington,  D.C 


cost  effective  simulator  selection  and  use  is  from  research  tasks  compris 


ing  a part  of  the  Aviation  Research  Laboratory  program  sponsored  by  the 


Air  Force  Office  of  Scientific  Research  under  Contract  No.  F44620 


Dr.  Charles  E.  Hutchinson  was  monitor  of  the  contract 


ACtRStMjS- 

«<n 

list 

1 duI.MOOWCT 


MMI 


SIMULATORS  FOR  TRAINING  AND  PROFIT 


Charles  0.  Hopkins 
Head,  Aviation  Research  Laboratory 
University  of  Illinois 
Urbana,  Illinois 


I ' 

I i 

I!  } 


t 

[ 

fc. 

» 

! 

T 

r 

L 


y 

II 


B 


ABSTRACT 

The  use  of  simulators  for  training  and  profit  is  discussed  in  terms  of  the  concept  of  cost 
effectiveness.  Increased  degree  and  fidelity  of  simulation  require  greater  equipment  complexity 
and  cost.  Data  are  presented  that  show  a high  negative  correlation  between  cost  and  field  reli- 
ability of  avionics  equipment.  There  is  a paucity  of  research  data  on  the  relationships  between 
simulator  fidelity  and  transfer  effectiveness.  The  results  of  the  first  and  only  recently  com- 
plete-1  experiment  to  investigate  transfer  of  initial  flight  training  as  a function  of  simulator 
cockpit  motion  are  summarized.  A rational  basis  for  simulator  selection  and  use  developed  by 
Jacobs  and  Roscoe  is  presented.  The  need  for  research  to  establish  relationships  between  trans- 
fer of  training  and  physical  characteristics  such  as  degree  and  fidelity  of  simulation  is  seen 
as  critical  to  the  widespread  future  use  of  simulators  for  training  and  profit. 


TRAINING,  PROFIT,  COST  EFFECTIVENESS 

The  motivation  for  building  and  using  air- 
craft simulators  in  the  future  can  be  for  training 
or  for  profit,  or  it  can  be  for  a combination  of 
training  and  profit.  The  primacy  of  one  or 
another  of  these  three  possibilities  will  deter- 
mine how  widespread  will  be  the  future  use  of 
simulators  in  flying  training  and  other  skills 
training. 

Historically,  training  has  been  an  Important 
concern  of  aircraft  simulator  purchasers  and  users. 
Profit  has  been  an  important  concern  of  aircraft 
simulator  builders  and  marketeers.  Unfortunately, 
with  few  notable  exceptions,  simulator  users  have 
shown  little  concern  for  the  "true"  profit,  if  any, 
that  can  be  derived  from  the  use  of  simulators  in 
training.  Many  have  been  fascinated  and  awed  by 
the  ever  increasingly  wonderful  technological 
developments  featured  by  the  marketeers  of  succes- 
sive generations  of  simulators.  On  the  other  hand, 
most  simulator  manufacturers  and  marketeers  have 
shown  little  concern  for  the  "true"  training  value 
of  ever  more  sophisticated  and  therefore  more 
costly  and  profitable  (to  them)  simulator  features. 

The  future  of  simulators  in  flying  training 
and  in  other  types  of  skills  training  can  be 
extremely  promising.  We  may  even  be  witnessing 
now  the  beginnings  of  a revolution  in  skills  train- 
ing of  all  kinds.  Heretofore  impossible,  this 
revolution* can  occur  now  because  of  recent 
behavioral  science  contributions  to  training 
technology  and  recent  engineering  contributions 
to  systems  simulation.  However,  the  fulfillment 
of  this  bright  promise  is  dependent  upon  the 
demonstration  of  cost  effectiveness  of  simulators 
in  skill  training  applications. 

"Cost  effectiveness,"  of  course,  is  merely  a 
concept  that  Incorporates  in  an  Important  and 
meaningful  way  our  familiar  goals  of  "training" 
and  "profit."  The  importance  of  this  concept  lies 
in  the  fact  that  the  profit  to  be  realized  refers 
to  the  training  program  rather  than  to  the  manufac- 
turer. This  is  not  to  say  that  cost  effective  uses 
of  simulators  will  not  also  be  profitable  to  manu- 
facturers. They  will  be.  Conversely,  however, 
unless  considerably  more  attention  is  given  to 


cost  effective  design  and  use  of  simulators  by 
users  and,  more  especially,  by  manufacturers,  the 
manufacturers'  profits  may  dry  up.  In  another 
context  I have  stated,  "Many  of  us  who  are  profes- 
sionally involved  with  the  use  of  simulators  in 
research  and  training  are  gravely  concerned  about 
the  effects  of  some  of  the  current  activities  in 
developing  and  selling  simulators.  The  acquisi- 
tion of  simulators  that  cost  several  times  as 
much  to  own  and  operate  as  their  counterpart 
airplanes  is  certain  to  produce  a backlash.  Such 
a reaction  will  set  back  the  desirable  use  of 
cost-effective  simulators  in  reasonable  research 
and  training  programs"  (Hopkins,  1975). 

We  know  quite  a lot  about  the  effectiveness 
of  simulators  in  training  programs.  Almost  any 
kind  of  simulator  can  be  used  effectively  in  a 
well-designed  training  program  with  specially 
trained  Instructors  and  highly  motivated  instruc- 
tors and  students.  We  can  train  effectively  in 
some  complex,  costly,  high  fidelity  simulators 
that  almost  approach  perfect  reproduction  of 
certain  of  the  aircraft  characteristics.  But 
we  also  can  train  effectively  in  quite  simple, 
inexpensive  simulators  that  may  amount  to  little 
more  than  static  mockups  of  the  cockpit  instru- 
ments and  controls.  We  can  even  train  effectively 
using  a photograph  of  the  cockpit  instrument  panel 
that  depicts  the  displays  and  controls.  Some 
primary  flight  training  programs  even  emphasize 
the  importance  of  "mental  rehearsal"  in  the 
absence  of  any  sensory  input  representing  the 
aircraft's  physical  characteristics  and  dynamic 
responses  as  an  effective  aid  in  training. 

In  spite  of  the  fact  that  we  know  that  more 
or  less  effective  training  can  be  accomplished 
with  a wide  variety  of  types  of  simulators  and 
related  training  devices,  we  don't  know  very  much 
about  how  some  of  the  specific  features  and 
characteristics  commonly  built  into  simulators 
contribute  to  or  detract  from  the  overall  effec- 
tiveness of  simulators  used  in  training  programs. 
The  important  issue  is  not  how  much  fidelity  of 
simulation  can  we  achieve.  The  issue  is  not  even 
one  of  how  much  fidelity  of  simulation  do  we  need 
(regardless  of  cost  in  dollars,  energy,  and  time). 
The  Important  issue  is  what  level (s)  of  fidelity 
and  degree  of  simulation  are  cost  effective. 
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Before  dealing  directly  with  a cost  effectiveness 
analysis  of  a specific  example  of  aircraft  simu- 
lator fidelity  let  us  consider  an  aspect  of  cost 
and  effectiveness  that  is  seldom  mentioned  In 
connection  with  simulators. 

FIDELITY,  COMPLEXITY,  COST,  RELIABILITY 

As  the  degree  and  fidelity  of  simulation 
increase,  the  engineering  complexity  and  the  costs 
of  the  simulator  rise  at  an  increasing  rate.  This 
has  implications  far  beyond  the  initial  costs  of 
complex  simulators.  One  of  these  is  perhaps  best 
illustrated  by  the  data  in  Figure  1 that  show  the 
relationship  between  unit  production  cost  and  field 
reliability  for  Air  Force  avionics  equipment. 


From  this  relationship,  field  reliability  of 
avionioB  can  be  crudely  predicted  (within  a factor 
of  3)  when  cost  is  known.  If,  for  example,  an 
equipment  costs  $ 100,000 , it  can  be  expected  to 
have  an  MFHBF  of  13  hours;  if  it  costs  $1,000  the 
expected  MFHBF  is  1,300  hours. 

The  direct  maintenance  and  repair  costs 
associated  with  low  mean  hours  between  failure 
for  a complex,  costly  simulator  can  be  estimated 
in  terms  of  dollars.  The  detrimental  effects 
upon  a training  program  of  frequent  disruptions 
due  to  equipment  failure  are  less  easily  estimated 
or  even  appropriately  expressed  in  terms  of  dollars. 

FIDELITY  AND  TRANSFER  EFFECTIVENESS 
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Figure  1.  Avionics  Field  Reliability  versus  Unit 
Production  Cost  (Adapted  from  Gates, 
et  al,  1974). 

These  data  points  were  drawn  from  a number  of 
sources  in  a study  performed  by  the  Institute  for 
Defense  Analyses  for  the  Defense  Advanced  Research 
Projects  Agency  (Gates,  Gourary,  Deitchman,  Rowan, 
and  Ueimer,  1974).  Although  these  data  are  not 
based  upon  simulators  they  are  based  upon  avionics 
equipments  that  are  comparable  to  simulators  in 
design  complexity,  type  of  components,  and  opera- 
tional use  and  maintenance.  The  coefficient  of 
correlation  between  reliability  and  cost  calculated 
from  the  data  points  in  Figure  1 is  r • -0.8667. 

The  linear  regression  accounts  for  75. 1%  of  the 
sample  variance. 

The  authors  of  the  report  state: 

As  complexity  increases , cost  increases  and 
reliability , as  measured  by  mean  flight  hours 
between  failures  (MFHBF)  decreases.  Thus, 

1 Figure  3]  shows  a median  relationship  in  which 

C 

MFHBF  m 1.3  x 10  /cost 


For  some  aircraft  simulator  features  we  know 
very  little  about  how  fidelity  of  simulation  is 
related  to  training  effectiveness.  Unfortunately, 
certain  of  these  features  can  be  among  the  most 
costly  ones  to  implement.  A prime  example  is  the 
case  of  simulator  motion. 

Transfer  of  training  experiments  have  been 
performed  in  experimental  psychological  research 
for  at  least  the  past  118  years.  Motion  systems 
of  varying  degrees  and  fidelity  of  simulation  have 
been  available  since  the  earliest  Link  trainers. 

In  spite  of  all  the  research  that  has  been  done 
on  transfer  of  training  from  ground-based  simu- 
lators to  aircraft  over  the  past  thirty  years,  the 
first  experiment  to  investigate  transfer  of 
primary  flight  training  from  a ground  based  simu- 
lator to  an  aircraft  as  a function  of  simulator 
motion  conditions  was  completed  only  one  year  ago 
(Jacobs  and  Roscoe,  1975).  This  research  confirmed 
the  results  of  the  many  experiments  and  informal 
observations  that  have  shown  ground-based  flight 
simulator  training  to  yield  positive  transfer  to 
performance  in  flight.  However,  differences  in 
transfer  were  not  significant  for  groups  of  pilots 
trained  with  no  simulator  motion,  normal-washout 
simulator  motion,  and  random-washout  simulator 
motion.  A summary  of  the  transfer  data  from  this 
experiment  is  shown  in  Table  1. 

TABLE  1 

(From  Jacobs  and  Roscoe,  1975) 

Mean  Times,  Trials,  and  Errors  to  Reach 
Performance  Criteria  in  the  Airplane,  Adjusted  to 
Eliminate  Individual  Aptitude  Effects,  for  a 
Control  Group  and  Three  Transfer  Groups  of  Nine 
Subjects  Each 


Control 

Group 

Cockpit  Motion 
Transfer  Group 

Airplane 

Only 

Normal 

Washout 

Fixed 

Base 

Random 

Washout 

Time  in  min 

182.4 

69.8 

80.0 

111.2 

Errors 

90.0 

46.5 

56.4 

59.9 

Trials 

38.5 

16.1 

17.1 

22.2 
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SIMULATOR  COST  EFFECTIVENESS 

Jacobs  and  Roscoe  (1975)  provide  an  extensive 
discussion  of  the  implications  of  these  results 
for  cost  effectiveness  of  ground  based  simulator 
training.  If  simulator  training  reduces  the  time 
required  for  aircraft  training  sufficiently  to  be 
cost  effective,  then  an  overall  training  program 
should  include  simulator  training.  Overall  train- 
ing cost  savings  are  realized  when  the  simulator 
operating  cost  is  less  than  aircraft  operating 
cost  by  a ratio  greater  than  the  inverse  of  the 
transfer  effectiveness  ratio.  The  transfer  effec- 
tiveness ratio  (Roscoe,  1971)  is  determined  as 
follows: 

T - T 

TER  - , where 


T • time  to  criterion  in  the  transfer  task  for 
the  control  group, 

T “ time  to  criterion  in  the  transfer  task  for 
the  experimental  group, 

Xt  - time  spent  on  the  practice  task  in  the  sim- 
ulator by  the  experimental  group. 

The  inverse  values  of  transfer  effectiveness 
ratios  for  the  different  simulator  motion  groups 
used  in  the  Jacobs  and  Roscoe  experiment  are  3.18 
for  normal  washout  motion,  3.35  for  no  motion,  and 
A. 00  for  random  washout  motion.  Multiplying  the 
inverse  transfer  effectiveness  ratio  of  3.18  by 
a typical  hourly  operating  cost  of  $15.30  for  a 
simple  sustained  pitch,  bank,  and  yaw  motion  simu- 
lator yields  a minimum  airplane  operating  cost  of 
$48.65  an  hour  for  economical  use  of  this  kind  of 
moving  base  simulator  for  training  in  the  flight 
curriculum  taught  in  this  experiment.  A similar 
calculation  based  on  a typical  hourly  operating 
cost  of  $10.60  for  a fixed-base  simulator  yielded 
a minimum  airplane  operating  cost  of  $35.44  an 
hour  for  economical  use  of  the  fixed-base  simu- 
lator. Since  the  typical  cost  of  operating  a 
primary  training  airplane  is  approximately  $28.00 
an  hour,  use  of  either  type  of  simulator  should 
be  rejected  as  uneconomical  if  there  were  no  other 
considerations.  However,  Jacobs  and  Roscoe  point 
out  a number  of  factors  operating  in  the  experi- 
ment that  served  to  limit  total  transfer  and 
transfer  effectiveness.  Nevertheless,  this  kind 
of  analysis  represents  the  approach  that  should 
be  taken  in  determining  the  cost  effectiveness  of 
a simulator  in  a training  program. 

Jacobs  and  Roscoe  also  developed  and  presented 
a rational  basis  for  simulator  selection  and  use 
as  follows: 


effectiveness  associated  with  better  conditions 


Figure  2.  Hypothetical  incremental  and  cumulative 
transfer  effectiveness,  in  a 6.5-hr 
initial  flight  training  curriculum,  as 
functions  (ITEF  and  CTEF)  of  the  amount 
of  training  time  in  representative 
fixed-base  and  moving-base  general 
aviation  flight  trainers  and  the  asso- 
ciated profit  or  loss. 

For  a particular  simulator,  a cost  effec- 
tiveness crossunder  point  is  reached  when  its 
incremental  transfer  effectiveness  ratio  equals 
the  ratio  of  its  hourly  cost  to  that  of  the 
counterpart  airplane.  With  cost  ratios  of  0.546 
and  0.379  between  the  two  simulators  and  the 
airplane  represented  in  Figure  2,  corresponding 
incremental  transfer  effectiveness  ratios  are 
reached  at  slightly  less  than  1 hr  and  2 hr, 
respectively,  for  this  brief,  6.5  hr  flight 
curriculum.  Thus,  in  each  cockpit  motion  con- 
dition, use  of  the  simulator  beyond  these  respec- 
tive points  would  waste  the  time  of  the  student, 
the  instructor,  and  the  simulator,  all  of  which 
may  be  expressed  in  terms  of  money. 

There  is  compelling  evidence  from  the  results 
obtained  that  the  amount  of  simulator  training 
given  students  in  this  experiment  was  uneconomical 
under  the  particular  circumstances  that  prevailed. 
For  a training  simulator  to  be  cost  effective,  its 
cost  must  be  low,  its  transfer  effectiveness  high, 
and  its  use  limited  to  the  point  at  which  its 
incremental  transfer  ratio  crosses  under  its  cost 
ratio  relative  to  the  airplane.  (Jacobs  and 
Roscoe,  1975). 


Figure  2 depicts  hypothetical  relationships 
among  incremental  and  cumulative  transfer  effec- 
tiveness and  associated  profit  or  loss  as  functions 
of  the  amount  of  training  time  in  representative 
fixed-base  and  moving-base  general  aviation  flight 
trainers.  The  scales  of  transfer,  time  and  cost 
have  been  set  to  be  consistent  with  the  amount  of 
training  and  findings  of  this  study,  but  the 
relationships  shown  are  of  a generalisable  nature, 
subject  to  scale  adjustments  to  accomodate  longer 
periods  of  training  and  higher  levels  of  transfer 


TRAINING,  PROFIT,  AND  RESEARCH 

Near  the  beginning  of  this  paper  it  was  stated 
that  the  fulfillment  of  the  bright  promise  for  use 
of  simulators  in  skills  training  is  dependent  upon 
the  demonstration  of  their  cost  effectiveness  in 
specific  applications.  Cost  effectiveness  can  be 
achieved  through  a series  of  steps  including 
specification  and  design  of  a simulator  for  the 
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purpose  for  which  it  is  to  be  used  and  then  using 
it  at  the  proper  phase  and  for  the  proper  duration 
in  the  training  program.  However,  the  decisions 
for  each  of  these  steps  must  be  based  upon  infor- 
mation derived  from  research.  Most  of  the  required 
research  has  not  been  done. 

If  simulators  are  to  be  widely  used  for  train- 
ing and  profit  in  the  future  a systematic  research 
program  must  be  conducted  to  establish  the  rela- 
tionships between  transfer  of  training  and  physical 
characteristics  such  as  degree  and  fidelity  of 
simulation. 
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